1. Field of the Invention
The present invention relates to a device designed to input data for controlling or remote-controlling, for example, an electronic apparatus.
2. Description of the Related Art
Most of the data input devices (hereinafter called "control devices") hitherto developed to input control data are hand-held ones. A user operates a control device of this type, while holding it in hand. Control devices most commonly used in households are remote controllers. A remote controller has several buttons on it. The user holds it in a hand, aims it at an electronic apparatus and pushes one of the buttons. When the button is pushed, the remote controller emits an infrared beam to the apparatus, thus giving instructions to the apparatus.
Among the control devices hitherto developed, other than the hand-held ones, is a so-called touch panel. The touch panel comprises a transparent or opaque panel. When a user pushes any part of the panel with a finger or an elongated object, the touch panel inputs a data item representing the pushing of that part or the position of that part. A transparent panel may be mounted on a display screen displaying images of buttons. In this case, the user may push that part of the transparent panel located above one of the buttons displayed, thereby to input the data item indicating that the button has been selected.
In the field of image recognition, research has been done for a method of recognizing hand signaling and gestures. No practically usable methods have been developed, however. This is because various requirements are imposed, which are hard to fulfill in practice. More specifically, it is required that the background on which to display hands or a person be monotonous so that image data may be processed more easily. It is also required that the hands or the person be displayed in a relatively large size at the central part of a display screen.
Also, research has been conducted to facilitate image recognition. More precisely, an attempt has been made to develop a method in which stereoscopic analysis is performed on an object, based on trigonometrical survey, in order to recognize the three-dimensional shape of the object. Nonetheless, only a few cases are known in which image recognition techniques are applied to remote controllers.
A user cannot operate a remote controller unless he or she holds it in a hand. It is undesirable that the remote controller is held in a hand which has gotten dirty.
To operate a touch panel, the user need not hold anything at all. However, the panel may get dirty as the user repeatedly touches the panel with fingers. It may become so dirty that the images display on the display screen can no longer be seen through the panel. If the panel is dirty after a long use by a user, other users are discouraged to touch the panel in order to input data.
In the technique of recognizing hand signaling, achieved by processing the image data, the results are hardly reliable, greatly influenced by noise. The technique should therefore be improved to eliminate the influence of noise.
There is a great demand for control devices which can be operated by bare hands, without being held in hand or attached to a display screen. However, no control devices of this type have been provided.
Pointing devices, such as mouses, have hitherto been commonly used to control personal computers and the like. A pointing device is a data input device which is operated to input position data to a computer. Various types of pointing devices are available. The most popular of the pointing device is the mouse. The mouse comprises a housing and a ball contained in the housing. The ball can rotate in any direction with respect to the housing.
In use, the mouse is electrically connected to the computer and placed on a desk. Once the mouse is put on the desk, the ball contacts the desk top at one point. As the mouse is moved back, forth, leftward or rightward on the desk, the ball rotates with respect to the mouse housing. The angles through which the ball rotate in two directions are detected, either mechanically or optically. The data representing the angles detected is supplied to the computer In the computer, this data is processed into a direction and a distance, for which the cursor is moved on the display screen.
Pointing devices other than the mouse have been put to use for inputting data to computers. Among them are a touch pad, a track ball, a small joystick, a touch panel and a pen device.
The mouse is easy to operate to input data to the computer. However, the mouse can hardly be used unless there is a flat wide surface such as the top of a desk. To operate his computer outdoors, by using the mouse, the user needs to find something having a flat wide surface. In a train or a car, it would be impossible for him to use the mouse to input data to the computer placed on his laps. Even if a desk is provided, the mouse cannot be used if the desktop is covered with books, folders or documents. To use the mouse, the desktop has to be cleared.
Portable computers, each having a pointing device other than the mouse, are now used in increasing numbers. The pointing device on the portable computer is a track ball, a touch pad, a small joystick or the like. The track ball is rotated with a finger. The direction and distance the track ball is rotated are detected, and the data representing them is input to the computer. The touch pad, available in various types, can detect the position of any part touched. When stroked with the user's finger, the touch pad generates data representing the direction and distance it is thus stroked.
A small joystick is shaped like a lever. It is provided on a keyboard. In most cases, it protrudes upwards, from among "G", "H" and "B" keys. When the joystick is rotated, the direction and angle by which it is rotated are detected. The direction and the angles are converted into two data items which represent the direction and distance for which the cursor is to move on the display screen. The data items are input to the computer.
Pointing accomplished by using a mouse, a track ball, a touch pad, or a small joystick proceeds in the following sequence of steps:
1. Detect the position the cursor assumes now. PA1 2. Determine a target position to which the cursor is to be moved. PA1 3. Determine the direction and distance for which the cursor is to be moved. PA1 4. Input data items representing the direction and the distance, thus determined. PA1 1. Determine a target position on the screen, to which the cursor is to be moved. PA1 2. Touch that part of the panel which is aligned with the target position on the screen. PA1 second point-motion detecting means for detecting that the finger tip is moved to select a menu item.
Steps 1 and 2 are interchangeable. Steps 3 and 4 may be repeated so that the cursor may keep moving in various directions in different directions.
The movement of the operator's hand for using the mouse differs from the movement of his hand for using other pointing devices. When the pointing is made by using the mouse, the position of the mouse approximately corresponds to the position of the cursor. Of course, when the mouse is lifted up and then moved, the relation between the mouse position and the cursor position is changed. Further, if the mouse driver having a function of accelerating the cursor is used, the mouse position does not completely correspond to the cursor position. However, from a view point of short distance operation, it can be said that the mouse position approximately corresponds to the cursor position. In other words, when moving the cursor, the operator may move his hand holding the mouse similar to the movement of the cursor.
By contrast, when a track ball, a touch pad or a small joystick is used to move the cursor on the display screen, the motion of the cursor is different from the motion of the user's hand.
In the case of a track ball, the direction and angle by which the ball is rotated correspond to the direction and distance for which the cursor moves, as long as the ball is rotated at a single touch to move the cursor for a relatively short distance. To move the cursor for a longer distance, the user rotates the ball more than two times, repeatedly moving the hand back and forth. Obviously, the motion of the cursor is no longer the same as the motion of the hand.
In the case of a touch pad, the motion of the user's finger stroking the pad is similar to the motion of the cursor on the display screen, provided that the cursor is moved by a single stroke on the pad to move the cursor for a relatively short distance. To move the cursor for a longer distance, the user strokes the pad two or more times, moving his hand back and forth. Hence, the motion of the cursor does not correspond to the motion of the hand.
In the case of a small joystick, the direction in which the stick is rotated and the time for which the stick is held in the rotated position correspond to the direction and distance for which the cursor moves. While the user's hand is holding the stick at a rotated position, the cursor keeps moving. The user must rotate the stick minutely to adjust the position of the cursor as long as the stick is rotated at a single touch to move the cursor for a relatively short distance. To move the cursor for a longer distance, the user must rotate the stick two times or more, by repeatedly moving the hand back and force. Thus, the motion of the cursor does not correspond to the motion of the hand. Therefore, the motion of the cursor does not correspond to the motion of the user's hand.
The difference between the mouse and any other pointing device with respect to the relationship between the motion of the cursor and that of the user's hand is conspicuous, particularly for Steps 3 and 4 of the pointing described above. In Steps 3 and 4, the cursor is first moved a long distance and then moved repeatedly, each time for a much shorter distance. The user can move the cursor a long distance at a fairly high speed by using a mouse if he has grasped the preset speed ratio between the cursor and the mouse.
To move the cursor a long distance by using a track ball or a touch pad, however, the user must rotate the ball or stroke the pad, several times. To move the cursor a long distance by using a small joystick or a touch pad, he must rotate the stick, keep watching the cursor moving on the screen and rotate the stick back as the cursor gets near the target.
To move the cursor a very short distance in a desired direction by using a mouse, the user only need to move the mouse in the same direction. When the user uses a touch pad to move the cursor in the same way, the user may stroke the pad in the same direction, but it is difficult for him to move his finger smoothly on the pad due to the friction between the pad and the finger tip. If the user touches the pad rather lightly in order to move the cursor minutely, his finger will likely leave the pad, inevitably making a clicking error. To move the cursor a very short distance in a desired direction, the user may operate a small joystick. In this cases the user must repeat slightly rotating the stick and rotating it back, each time to move the cursor a little. It takes a long time to move the cursor, bit by bit.
Generally, the mouse excels in operability but is disadvantageous in terms of portability, whereas the touch pad, track ball and small joystick are quite portable but inferior in operability.
The user a touch panel in place of a mouse, to perform pointing. If a transparent touch panel is mounted on the display screen, the user can designate any desired point on the screen, merely by touching that part of the panel which is aligned with the desired point of the screen.
Pointing accomplished by using a touch panel proceeds in only the following two steps, far less than in the pointing achieved by means of a mouse, a track ball, a touch pad, or a small joystick
However, some problems arise when a touch panel is used to accomplish pointing. First, the user's finger touching the panel conceals that part of the displayed data which is located at the target position. Second, it is impossible to point anything displayed that is smaller than the finger tip. Third, the display screen gets dirty as it is repeatedly touched with fingers. Further, to drag a menu item on the display screen, the user needs to stroke the touch panel with his finger. The user is discouraged from stroking the touch panel, however, in fear of soiling the panel or feeling frictional touch on the panel.
The user may touch the panel, selecting any desired menu item and then touch the panel again at a different part thereof, in order to display the selected icon at that position on the screen which is aligned with the part of the panel he has touched again. If this is the case, the menu item does not appear as if being dragged to that position on the screen. This may be why touch panels are attached to the display screens of portable computers, though they are used as the operation panels of ATMs (installed at banks) which display large buttons and keys or are provided on word processors of a particular type.
Some touch panels are designed to be touched with a pen. They are more popular than any other pointing device, as a device for inputting data into personal electronic apparatuses. They have been commercialized as an input interface of pen computers and PDAs (Personal Digital Assistants). When touched with a pen, the touch panel can select smaller menu items and gets less dirty than when touched with fingers, and can easily drag the selected menu item to any desired position on the display screen. In addition, the user can input data representing hand-written characters by writing the characters with the pen on the touch panel.
To use the pen in combination with the touch panel, the user has to hold the pen in a hand. This does not matter if the panel is attached to a PDA. When the touch panel is used along with a keyboards as in the case of inputting data into a portable computers it would be cumbersome for the user to lift one hand off the keyboard and then hold the pen instead in this hand, in order to point any desired item displayed on the screen. Even in the case of a touch panel on a PDA, the user may feel it troublesome to take the pen to point a desired item displayed on the screens though he may not feel it so troublesome when he moves the pen to input characters into the portable computer. Being a small item, the pen is likely to be lost. The pen may be tied to the touch panel by a string. If the pen is tied to the panel, however, it becomes difficult for the user to use the pen.
The user may designate any desired item displayed on the screen, without using a pointing device of any type. For example, a data input device may be used which processes the image data representing the user's hand, thereby detecting the shape and position of the user's hand. Such a data input device, if any, enables the user to designate a desired one of items displayed on the screen, only by pointing the desired item with a finger. As a result, the motion of the cursor is as faithful to the motion of the finger, as to the motion of a mouse. Needless to say, it is far easier to move a finger than to operate a touch pad, a track ball or a small joystick, in order to point any desired item on the display screen. In addition, no flat wide surface is required as in the case wherein a mouse is used to point any item displayed on the screen. The user need not touch the display screen, and the display screen can remain clean. Nor does the user have to hold a pen device in the hand.
Any desired item displayed on the screen may be designated by another method, without using a pointing device of any type. In this method, a color marker is used to mark a part of the user's hand, and the color-marked part of the hand is detected from an image of the hand. The method is widely employed by various apparatuses for determining and analyzing the motions of objects. It can be applied to pointing devices for use in combination with portable computers. However, it cannot be practiced without using expensive image-processing hardware. Nor can it work well if any garment the user wears happens to be of the same color as the marker Further, the user may not be willing to have his hand marked by the color marker. In view of these disadvantages, the method hardly seems practical.
Pointing systems have been made on a research basis, each recognizing the user's bare hand and the position of the tip of any finger, from an image of the user's hand. These pointing systems do not work well in most cases, because it is usually very difficult to distinguish the image of the hand from the images of other moving objects.
As has been mentioned, every pointing technique developed hitherto is advantageous in one respect and disadvantageous in another. It can hardly be said that a pointing device is available which functions well in combination with portable computers, even when the computers are used outdoors.